Memory modules with reliability and serviceability functions

ABSTRACT

One embodiment of the present invention sets forth a memory module that includes at least one memory chip, and an intelligent chip coupled to the at least one memory chip and a memory controller, where the intelligent chip is configured to implement at least a part of a RAS feature. The disclosed architecture allows one or more RAS features to be implemented locally to the memory module using one or more intelligent register chips, one or more intelligent buffer chips, or some combination thereof. Such an approach not only increases the effectiveness of certain RAS features that were available in prior art systems, but also enables the implementation of certain RAS features that were not available in prior art systems.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application is a continuation of U.S. patent applicationSer. No. 13/280,251, filed Oct. 24, 2011, which is continuation of U.S.patent application Ser. No. 11/763,365, which was filed Jun. 14, 2007and which was a continuation-in part of U.S. patent application Ser. No.11/474,076, filed on Jun. 23, 2006, which claims priority to U.S.Provisional Patent Application No. 60/693,631, filed on Jun. 24, 2005.U.S. patent application Ser. No. 11/763,365 also was acontinuation-in-part of U.S. patent application Ser. No. 11/515,223,filed on Sep. 1, 2006, which claims priority to U.S. Provisional PatentApplication No. 60/713,815, filed on Sep. 2, 2005. U.S. patentapplication Ser. No. 11/763,365 also claimed the priority benefit ofU.S. Provisional Patent Application No. 60/814,234, filed on Jun. 16,2006 and titled, “Memory Systems and Memory Modules.” The subject matterof the above-listed related applications is hereby incorporated hereinby reference.

BACKGROUND OF THE INVENTION

1. Field of Invention

This application relates generally to memory systems and morespecifically to memory modules that include reliability andserviceability features and functionalities.

2. Description of the Related Art

Memory systems in computers are typically built from memory modules. Anexample of a common type of memory module is a Dual Inline Memory Moduleor DIMM. It is very important for many computers, particularly inclasses of computers known as servers, that the memory system be asreliable as possible. It is also very important that the computer andmemory system are built in such a way that the computer can be servicedas quickly, easily and as inexpensively as possible. Together theseimportant features are often known as Reliability and Serviceability(RAS) features.

SUMMARY OF THE INVENTION

Computers are often required to have a very low probability of failure.In order to reduce the probability of failure it is often required toadd reliability features to the memory system and to the memory modulesin a computer memory system.

The embodiments described herein are a series of RAS features that maybe used in memory systems. One embodiment of the present invention setsforth a memory module that includes at least one memory chip, and anintelligent chip coupled to the at least one memory chip and a memorycontroller, where the intelligent chip is configured to implement atleast a part of a RAS feature.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A depicts connectivity in an embodiment that includes anintelligent register and multiple buffer chips.

FIG. 1B depicts a generalized layout of components on a DIMM, includingLEDs.

FIG. 2A depicts a memory subsystem with a memory controller incommunication with multiple DIMMs.

FIG. 2B depicts a side view of a stack of memory including anintelligent buffer chip.

FIG. 3 depicts steps for performing a sparing substitution.

FIG. 4 depicts a memory subsystem where a portion of the memory on aDIMM is spared.

FIG. 5 depicts a selection of functions optionally implemented in anintelligent register chip or an intelligent buffer chip.

FIG. 6A depicts a memory stack in one embodiment with eight memory chipsand one intelligent buffer.

FIG. 6B depicts a memory stack in one embodiment with nine memory chipsand one intelligent buffer.

FIG. 7A depicts an embodiment of a DIMM implementing checkpointing.

FIG. 7B depicts an depicts an exploded view of an embodiment of a DIMMimplementing checkpointing.

FIG. 8A depicts adding a memory chip to a memory stack.

FIG. 8B depicts adding a memory stack to a DIMM.

FIG. 8C depicts adding a DIMM to another DIMM.

FIG. 9A depicts a memory subsystem that uses redundant signal paths.

FIG. 9B a generalized bit field for communicating data.

FIG. 9C depicts the bit field layout of a multi-cycle packet.

FIG. 9D depicts examples of bit fields for communicating data.

DETAILED DESCRIPTION

In order to build cost-effective memory modules it can be advantageousto build register and buffer chips that do have the ability to performlogical operations on data, dynamic storage of information, manipulationof data, sensing and reporting or other intelligent functions. Suchchips are referred to in this specification as intelligent registerchips and intelligent buffer chips. The generic term, “intelligentchip,” is used herein to refer to either of these chips. Intelligentregister chips in this specification are generally connected between thememory controller and the intelligent buffer chips. The intelligentbuffer chips in this specification are generally connected between theintelligent register chips and one or more memory chips. One or more RASfeatures may be implemented locally to the memory module using one ormore intelligent register chips, one or more intelligent buffer chips,or some combination thereof.

In the arrangement shown in FIG. 1A, one or more intelligent registerchips 102 are in direct communication with the host system 104 via theaddress, control, clock and data signals to/from the host system. One ormore intelligent buffer chips 107A-107D are disposed between theintelligent register chips and the memory chips 106A-106D. The signals110, 111, 112, 113, 118 and 119 between an intelligent register chip andone or more intelligent buffer chips may be shared by the one or moreintelligent buffer chips. In the embodiment depicted, the signals fromthe plural intelligent register chips to the intelligent buffer chipsand, by connectivity, to the plural memory chips, may be independentlycontrollable by separate instances of intelligent register chips. Inanother arrangement the intelligent buffer chips are connected to astack of memory chips.

The intelligent buffer chips may buffer data signals and/or addresssignals, and/or control signals. The buffer chips 107A-107D may beseparate chips or integrated into a single chip. The intelligentregister chip may or may not buffer the data signals as is shown in FIG.1A.

The embodiments described here are a series of RAS features that may beused in memory systems. The embodiments are particularly applicable tomemory systems and memory modules that use intelligent register andbuffer chips.

Indication of Failed Memory

As shown in FIG. 1B, light-emitting diodes (LEDs) 108, 109 can bemounted on a memory module 100. The CPU or host or memory controller, oran intelligent register can recognize or determine if a memory chip106A-106J on a memory module has failed and illuminate one or more ofthe LEDs 108, 109. If the memory module contains one or more intelligentbuffer chips 107A, 107H or intelligent register chips 102, these chipsmay be used to control the LEDs directly. As an alternative to the LEDsand in combination with the intelligent buffer and/or register chips,the standard non-volatile memory that is normally included on memorymodules to record memory parameters may be used to store information onwhether the memory module has failed.

In FIG. 1B, the data signals are not buffered (by an intelligentregister chip or by an intelligent buffer chip). Although theintelligent buffer chips 107A-107H are shown in FIG. 1B as connecteddirectly to the intelligent register chip and act to buffer signals fromthe intelligent register chip, the same or other intelligent bufferchips may also be connected to buffer the data signals.

Currently indication of a failed memory module is done indirectly if itis done at all. One method is to display information on the failedmemory module on a computer screen. Often only the failing logicalmemory location is shown on a screen, perhaps just the logical addressof the failing memory cell in a DRAM, which means it is very difficultfor the computer operator or repair technician to quickly and easilydetermine which physical memory module to replace. Often the computerscreen is also remote from the physical location of the memory moduleand this also means it is difficult for an operator to quickly andeasily find the memory module that has failed. Another current methoduses a complicated and expensive combination of buttons, panels,switches and LEDs on the motherboard to indicate that a component on orattached to the motherboard has failed. None of these methods place theLED directly on the failing memory module allowing the operator toeasily and quickly identify the memory module to be replaced. Thisembodiment adds just one low-cost part to the memory module.

This embodiment is part of the memory module and thus can be used in anycomputer. The memory module can be moved between computers of differenttypes and manufacturer.

Further, the intelligent register chip 102 and/or buffer chip 107A-107Jon a memory module can self-test the memory and indicate failure byilluminating an LED. Such a self-test may use writing and reading of asimple pattern or more complicated patterns such as, for example,“walking-1's” or “checkerboard” patterns that are known to exercise thememory more thoroughly. Thus the failure of a memory module can beindicated via the memory module LED even if the operating system orcontrol mechanism of the computer is incapable of working.

Further, the intelligent buffer chip and/or register chip on a memorymodule can self-test the memory and indicate correct operation viaillumination of a second LED 109. Thus a failed memory module can beeasily identified using the first LED 108 that indicates failure andswitched by the operator with a replacement. The first LED might be redfor example to indicate failure. The memory module then performs aself-test and illuminates the second LED 109. The second LED might begreen for example to indicate successful self-test. In this manner theoperator or service technician can not only quickly and easily identifya failing memory module, even if the operating system is not working,but can effect a replacement and check the replacement, all without theintervention of an operating system.

Memory Sparing

One memory reliability feature is known as memory sparing.

Under one definition, the failure of a memory module occurs when thenumber of correctable errors caused by a memory module reaches a fixedor programmable threshold. If a memory module or part of a memory modulefails in such a manner in a memory system that supports memory sparing,another memory module can be assigned to take the place of the failedmemory module.

In the normal mode of operation, the computer reads and writes data toactive memory modules. In some cases, the computer may also containspare memory modules that are not active. In the normal mode ofoperation the computer does not read or write data to the spare memorymodule or modules, and generally the spare memory module or modules donot store data before memory sparing begins. The memory sparing functionmoves data from the memory module that is showing errors to the sparememory modules if the correctable error count exceeds the thresholdvalue. After moving the data, the system inactivates the failed memorymodule and may report or record the event.

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful memory sparing capabilities may be implemented.

For example, and as illustrated in FIG. 2A the intelligent register chip242 that is connected indirectly or directly to all DRAM chips 243 on amemory module 250 may monitor temperature of the DIMM, the buffer chipsand DRAM, the frequency of use of the DRAM and other parameters that mayaffect failure. The intelligent register chip can also gather data aboutall DRAM chip failures on the memory module and can make intelligentdecisions about sparing memory within the memory module instead ofhaving to spare an entire memory module.

Further, as shown in FIG. 2A and FIG. 2B, an intelligent buffer chip 247that may be connected to one or more DRAMs 245 in a stack 200 is able tomonitor each DRAM 245 in the stack and if necessary spare a DRAM 246 inthe stack. In the exemplary embodiment, the spared DRAM 246 is shown asan inner component of the stack. In other possible embodiments thespared DRAM may be any one of the components of the stack includingeither or both of the top and bottom DRAMs.

Although the intelligent buffer chips 247 are shown in FIG. 2B asconnected directly to the intelligent register chip 242 and to buffersignals from the intelligent register chip, the same or otherintelligent buffer chips may also be connected to buffer the datasignals. Thus, by including intelligent register and buffer chips in amemory module, it is possible to build memory modules that can implementmemory sparing at the level of being able to use a spare individualmemory, a spare stack of memory, or a spare memory module.

In some embodiments, and as shown in FIG. 3, a sparing method 380 may beimplemented in conjunction with a sparing strategy. In such a case, theintelligent buffer chip may calculate replacement possibilities 382,optimize the replacement based on the system 384 or a given strategy andknown characteristics of the system, advise the host system of thesparing operation to be performed 386, and perform the sparingsubstitution or replacement 388.

Memory Mirroring

Another memory reliability feature is known as memory mirroring.

In normal operation of a memory mirroring mode, the computer writes datato two memory modules at the same time: a primary memory module (themirrored memory module) and the mirror memory module.

If the computer detects an uncorrectable error in a memory module, thecomputer will re-read data from the mirror memory module. If thecomputer still detects an uncorrectable error, the computer system mayattempt other means of recovery beyond the scope of simple memorymirroring. If the computer does not detect an error, or detects acorrectable error, from the mirror module, the computer will accept thatdata as the correct data. The system may then report or record thisevent and proceed in a number of ways (including returning to check theoriginal failure, for example).

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful memory mirroring capabilities may be implemented.

For example, as shown in FIG. 4, the intelligent register chip 442allows a memory module to perform the function of both mirrored andmirror modules by dividing the DRAM on the module into two sections 460and 470. The intelligent buffer chips may allow DRAM stacks to performboth mirror and mirrored functions. In the embodiment shown in FIG. 4,the computer or the memory controller 400 on the computer motherboardmay still be in control of performing the mirror functions by readingand writing data to as if there were two memory modules.

In another embodiment, a memory module with intelligent register chips442 and/or intelligent buffer chips 447 that can perform mirroringfunctions may be made to look like a normal memory module to the memorycontroller. Thus, in the embodiment of FIG. 4, the computer is unawarethat the module is itself performing memory mirroring. In this case, thecomputer may perform memory sparing. In this manner both memory sparingand memory mirroring may be performed on a computer that is normally notcapable of providing mirroring and sparing at the same time.

Other combinations are possible. For example a memory module withintelligent buffer and/or control chips can be made to perform sparingwith or without the knowledge and/or support of the computer. Thus thecomputer may, for example, perform mirroring operations while the memorymodule simultaneously provides sparing function.

Although the intelligent buffer chips 447 are shown in FIG. 4 asconnected directly to the intelligent register chip 442 and to buffersignals from the intelligent register chip, the same or otherintelligent buffer chips may also be connected to buffer the datasignals.

Memory RAID

Another memory reliability feature is known as memory RAID.

To improve the reliability of a computer disk system it is usual toprovide a degree of redundancy using spare disks or parts of disks in adisk system known as Redundant Array of Inexpensive Disks (RAID). Thereare different levels of RAID that are well-known and correspond todifferent ways of using redundant disks or parts of disks. In manycases, redundant data, often parity data, is written to portions of adisk to allow data recovery in case of failure. Memory RAID improves thereliability of a memory system in the same way that disk RAID improvesthe reliability of a disk system. Memory mirroring is equivalent tomemory RAID level 1, which is equivalent to disk RAID level 1.

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful memory RAID capabilities may be implemented.

For example, as shown in FIG. 4, the intelligent register chip 442 on amemory module allows portions of the memory module to be allocated forRAID operations. The intelligent register chip may also include thecomputation necessary to read and write the redundant RAID data to aDRAM or DRAM stack allocated for that purpose. Often the parity data iscalculated using a simple exclusive-OR (XOR) function that may simply beinserted into the logic of an intelligent register or buffer chipwithout compromising performance of the memory module or memory system.

In some embodiments, portions 460 and 470 of the total memory on amemory module 450 are allocated for RAID operations. In otherembodiments, the portion of the total memory on the memory module thatis allocated for RAID operations may be a memory device on a DIMM 243 ora memory device in a stack 245.

In some embodiments, physically separate memory modules 451, and 452 ofthe total memory in a memory subsystem are allocated for RAIDoperations.

Memory Defect Re-Mapping

One of the most common failure mechanisms for a memory system is for aDRAM on a memory module to fail. The most common DRAM failure mechanismis for one or more individual memory cells in a DRAM to fail or degrade.A typical mechanism for this type of failure is for a defect to beintroduced during the semiconductor manufacturing process. Such a defectmay not prevent the memory cell from working but renders it subject topremature failure or marginal operation. Such memory cells are oftencalled weak memory cells. Typically this type of failure may be limitedto only a few memory cells in array of a million (in a 1 Mb DRAM) ormore memory cells on a single DRAM. Currently the only way to prevent orprotect against this failure mechanism is to stop using an entire memorymodule, which may consist of dozens of DRAM chips and contain a billion(in a 1 Gb DIMM) or more individual memory cells. Obviously the currentstate of the art is wasteful and inefficient in protecting againstmemory module failure.

In a memory module that uses intelligent buffer or intelligent registerchips, it is possible to locate and/or store the locations of weakmemory cells. A weak memory cell will often manifest its presence byconsistently producing read errors. Such read errors can be detected bythe memory controller, for example using a well-known Error CorrectionCode (ECC).

In computers that have sophisticated memory controllers, certain typesof read errors can be detected and some of them can be corrected. Indetecting such an error the memory controller may be designed to notifythe DIMM of both the fact that a failure has occurred and/or thelocation of the weak memory cell. One method to perform thisnotification, for example, would be for the memory controller to writeinformation to the non-volatile memory or SPD on a memory module. Thisinformation can then be passed to the intelligent register and/or bufferchips on the memory module for further analysis and action. For example,the intelligent register chip can decode the weak cell locationinformation and pass the correct weak cell information to the correctintelligent buffer chip attached to a DRAM stack.

Alternatively the intelligent buffer and/or register chips on the memorymodule can test the DRAM and detect weak cells in an autonomous fashion.The location of the weak cells can then be stored in the intelligentbuffer chip connected to the DRAM.

Using any of the methods that provide information on weak cell location,it is possible to check to see if the desired address is a weak memorycell by using the address location provided to the intelligent bufferand/or register chips. The logical implementation of this type oflook-up function using a tabular method is well-known and the table usedis often called a Table Lookaside Buffer (TLB), Translation LookasideBuffer or just Lookaside Buffer. If the address is found to correspondto a weak memory cell location, the address can be re-mapped using a TLBto a different known good memory cell. In this fashion the TLB has beenused to map-out or re-map the weak memory cell in a DRAM. In practice itmay be more effective or efficient to map out a row or column of memorycells in a DRAM, or in general a region of memory cells that include theweak cell. In another embodiment, memory cells in the intelligent chipcan be distributed for the weak cells in the DRAM.

FIG. 5 shows an embodiment of an intelligent buffer chip or intelligentregister chip which contains a TLB 560 and a store 580 for a mappingfrom weak cells to known good memory cells.

Memory Status and Information Reporting

There are many mechanisms that computers can use to increase their ownreliability if they are aware of status and can gather information aboutthe operation and performance of their constituent components. As anexample, many computer disk drives have Self Monitoring Analysis andReporting Technology (SMART) capability. This SMART capability gathersinformation about the disk drive and reports it back to the computer.The information gathered often indicates to the computer when a failureis about to occur, for example by monitoring the number of errors thatoccur when reading a particular area of the disk.

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful self-monitoring and reporting capabilities may beimplemented.

Information such as errors, number and location of weak memory cells,and results from analysis of the nature of the errors can be stored in astore 580 and can be analyzed by an analysis function 590 and/orreported to the computer. In various embodiments, the store 580 and theanalysis function 590 can be in the intelligent buffer and/or registerchips. Such information can be used either by the intelligent bufferand/or register chips, by an action function 570 included in theintelligent buffer chip, or by the computer itself to take action suchas to modify the memory system configuration (e.g. sparing) or alert theoperator or to use any other mechanism that improves the reliability orserviceability of a computer once it is known that a part of the memorysystem is failing or likely to fail.

Memory Temperature Monitoring and Thermal Control

Current memory system trends are towards increased physical density andincreased power dissipation per unit volume. Such density and powerincreases place a stress on the thermal design of computers. Memorysystems can cause a computer to become too hot to operate reliably. Ifthe computer becomes too hot, parts of the computer may be regulated orperformance throttled to reduce power dissipation.

In some cases a computer may be designed with the ability to monitor thetemperature of the processor or CPU and in some cases the temperature ofa chip on-board a DIMM. In one example, a Fully-Buffered DIMM orFB-DIMM, may contain a chip called an Advanced Memory Buffer or AMB thathas the capability to report the AMB temperature to the memorycontroller. Based on the temperature of the AMB the computer may decideto throttle the memory system to regulate temperature. The computerattempts to regulate the temperature of the memory system by reducingmemory activity or reducing the number of memory reads and/or writesperformed per unit time. Of course by measuring the temperature of justone chip, the AMB, on a memory module the computer is regulating thetemperature of the AMB not the memory module or DRAM itself.

In a memory module that includes intelligent register and/or intelligentbuffer chips, more powerful temperature monitoring and thermal controlcapabilities may be implemented.

For example if a temperature monitoring device 595 is included into anintelligent buffer or intelligent register chip, measured temperaturecan be reported. This temperature information provides the intelligentregister chips and/or the intelligent buffer chips and the computer muchmore detailed and accurate thermal information than is possible inabsence of such a temperature monitoring capability. With more detailedand accurate thermal information, the computer is able to make betterdecisions about how to regulate power or throttle performance, and thistranslates to better and improved overall memory system performance fora fixed power budget.

As in the example of FIG. 6A, the intelligent buffer chip 610 may beplaced at the bottom of a stack of DRAM chips 630A. By placing theintelligent buffer chip in close physical proximity and also closethermal proximity to the DRAM chip or chips, the temperature of theintelligent buffer chip will accurately reflect the temperature of theDRAM chip or chips. It is the temperature of the DRAM that is the mostimportant temperature data that the computer needs to make betterdecisions about how to throttle memory performance. Thus, the use of atemperature sensor in an intelligent buffer chip greatly improves thememory system performance for a fixed power budget Further theintelligent buffer chip or chips may also report thermal data to anintelligent register chip on the memory module. The intelligent bufferchip is able to make its own thermal decisions and steer, throttle,re-direct data or otherwise regulate memory behavior on the memorymodule at a finer level of control than is possible by using the memorycontroller alone.

Memory Failure Reporting

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful memory failure reporting may be implemented.

For example, memory failure can be reported, even in computers that usememory controllers that do not support such a mechanism, by using theError Correction Coding (ECC) signaling as described in thisspecification.

ECC signaling may be implemented by deliberately altering one or moredata bits such that the ECC check in the memory controller fails.

Memory Access Pattern Reporting and Performance Control

The patterns of operations that occur in a memory system, such as reads,writes and so forth, their frequency distribution with time, thedistribution of operations across memory modules, and the memorylocations that are addressed, are known as memory system accesspatterns. In the current state of the art, it is usual for a computerdesigner to perform experiments across a broad range of applications todetermine memory system access patterns and then design the memorycontroller of a computer in such a way as to optimize memory systemperformance. Typically, a few parameters that are empirically found tomost affect the behavior and performance of the memory controller may beleft as programmable so that the user may choose to alter theseparameters to optimize the computer performance when using a particularcomputer application. In general, there is a very wide range of memoryaccess patterns generated by different applications, and, thus, a verywide range of performance points across which the memory controller andmemory system performance must be optimized. It is therefore impossibleto optimize performance for all applications. The result is that theperformance of the memory controller and the memory system may be farfrom optimum when using any particular application. There is currentlyno easy way to discover this fact, no way to easily collect detailedmemory access patterns while running an application, no way to measureor infer memory system performance, and no way to alter, tune or in anyway modify those aspects of the memory controller or memory systemconfiguration that are programmable.

Typically a memory system that comprises one or more memory modules isfurther subdivided into ranks (typically a rank is thought of as a setof DRAM that are selected by a single chip select or CS signal), theDRAM themselves, and DRAM banks (typically a bank is a sub-array ofmemory cells inside a DRAM). The memory access patterns determine howthe memory modules, ranks, DRAM chips and DRAM banks are accessed forreading and writing, for example. Access to the ranks, DRAM chips andDRAM banks involves turning on and off either one or more DRAM chips orportions of DRAM chips, which in turn dissipates power. This dissipationof power caused by accessing DRAM chips and portions of DRAM chipslargely determines the total power dissipation in a memory system. Powerdissipation depends on the number of times a DRAM chip has to be turnedon or off or the number of times a portion of a DRAM chip has to beaccessed followed by another portion of the same DRAM chip or anotherDRAM chip. The memory access patterns also affect and determineperformance. In addition, access to the ranks, DRAM chips and DRAM banksinvolves turning on and off either whole DRAM chips or portions of DRAMchips, which consumes time that cannot be used to read or write data,thereby negatively impacting performance.

In the compute platforms used in many current embodiments, the memorycontroller is largely ignorant of the effect on power dissipation orperformance for any given memory access or pattern of access.

In a memory module that includes intelligent register and/or intelligentbuffer chips, however, powerful memory access pattern reporting andperformance control capabilities may be implemented.

For example an intelligent buffer chip with an analysis block 590 thatis connected directly to an array of DRAMs is able to collect andanalyze information on DRAM address access patterns, the ratio of readsto writes, the access patterns to the ranks, DRAM chips and DRAM banks.This information may be used to control temperature as well asperformance. Temperature and performance may be controlled by alteringtiming, power-down modes of the DRAM, and access to the different ranksand banks of the DRAM. Of course, the memory system or memory module maybe sub-divided in other ways.

Check Coding at the Byte Level

Typically, data protection and checking is provided by adding redundantinformation to a data word in a number of ways. In one well-knownmethod, called parity protection, a simple code is created by adding oneor more extra bits, known as parity bits, to the data word. This simpleparity code is capable of detecting a single bit error. In anotherwell-known method, called ECC protection, a more complex code is createdby adding ECC bits to the data word. ECC protection is typically capableof detecting and correcting single-bit errors and detecting, but notcorrecting, double-bit errors. In another well-known method calledChipKill, it is possible to use ECC methods to correctly read a dataword even if an entire chip is defective. Typically, these correctionmechanisms apply across the entire data word, usually 64 or 128 bits (ifECC is included, for example, the data word may be 72 or 144 bits,respectively).

DRAM chips are commonly organized into one of a very few configurationsor organizations. Typically, DRAMs are organized as x4, x8, or x16;thus, four, eight, or 16 bits are read and written simultaneously to asingle DRAM chip.

In the current state of the art, it is difficult to provide protectionagainst defective chips for all configurations or organizations of DRAM.

In a memory module that includes intelligent register and/or intelligentbuffer, chips powerful check coding capabilities may be implemented.

For example, as shown in FIG. 6B, using an intelligent buffer chip 610connected to a stack of x8 DRAMs 630B checking may be performed at thebyte level (across 8 bits), rather than at the data word level. Onepossibility, for example, is to include a ninth DRAM 620, rather thaneight DRAMs, in a stack and use the ninth DRAM for check codingpurposes.

Other schemes can be used that give great flexibility to the type andform of the error checking. Error checking may not be limited to simpleparity and ECC schemes, other more effective schemes may be used andimplemented on the intelligent register and/or intelligent buffer chipsof the memory module. Such effective schemes may include block andconvolutional encoding or other well-known data coding schemes. Errorsthat are found using these integrated coding schemes may be reported bya number of techniques that are described elsewhere in thisspecification. Examples include the use of ECC Signaling.

Checkpointing

In High-Performance Computing (HPC), it is typical to connect largenumbers of computers in a network, also sometimes referred to as acluster, and run applications continuously for a very long time usingall of the computers (possibly days or weeks) to solve very largenumerical problems. It is therefore a disaster if even a single computerfails during computation.

One solution to this problem is to stop the computation periodically andsave the contents of memory to disk. If a computer fails, thecomputation can resume from the last saved point in time. Such aprocedure is known as checkpointing. One problem with checkpointing isthe long period of time that it takes to transfer the entire memorycontents of a large computer cluster to disk.

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful checkpointing capabilities may be implemented.

For example, an intelligent buffer chip attached to stack of DRAM canincorporate flash or other non-volatile memory. The intelligent registerand/or buffer chip can under external or autonomous command instigateand control the checkpointing of the DRAM stack to flash memory.Alternatively, one or more of the chips in the stack may be flash chipsand the intelligent register and/or buffer chips can instigate andcontrol checkpointing one or more DRAMs in the stack to one or moreflash chips in the stack.

In the embodiment shown in the views of FIG. 7A and FIG. 7B, the DIMMPCB 710 is populated with a stacks of DRAM S0-S8 on one side and stacksof flash S9-S17, on the other side, where each flash memory in a flashstack corresponds with one of the DRAM in the opposing DRAM stack. Undernormal operation, the DIMM uses only the DRAM circuits—the flash devicesmay be unused, simply in a ready state. However, upon a checkpointevent, memory contents from the DRAMs are copied by the intelligentregister and/or buffer chips to their corresponding Flash memories. Inother implementations, the flash chips do not have to be in a stackorientation.

Read Retry Detection

In high reliability computers, the memory controller may supports errordetection and error correction capabilities. The memory controller maybe capable of correcting single-bit errors and detecting, but typicallynot correcting, double-bit errors in data read from the memory system.When such a memory controller detects a read data error, it may also beprogrammed to retry the read to see if an error still occurs. If theread data error does occur again, there is likely to be a permanentfault, in which case a prescribed path for either service oramelioration of the problem can be followed. If the error does not occuragain, the fault may be transient and an alternative path may be taken,which might consist solely of logging the error and proceeding asnormal. More sophisticated retry mechanisms can be used if memorymirroring is enabled, but the principles described here remain the same.

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful read retry detection capabilities may beimplemented. Such a memory module is also able to provide read retrydetection capabilities for any computer, not just those that have aspecial-purpose and expensive memory controllers.

For example, the intelligent register and/or buffer chips can beprogrammed to look for successive reads to memory locations without anintervening write to that same location. In systems with a cache betweenthe processor and memory system, this is an indication that the memorycontroller is retrying the reads as a result of seeing an error. In thisfashion, the intelligent buffer and/or register chips can monitor theerrors occurring in the memory module to a specific memory location, toa specific region of a DRAM chip, to a specific bank of a DRAM or anysuch subdivision of the memory module. With this information, theintelligent buffer and/or register chip can make autonomous decisions toimprove reliability (such as making use of spares) or report the detailsof the error information back to the computer, which can also makedecisions to improve reliability and serviceability of the memorysystem.

In some embodiments, a form of retry mechanism may be employed in a datacommunication channel. Such a retry mechanism is used to catch errorsthat occur in transmission and ask for an incomplete or incorrecttransmission to be retried. The intelligent buffer and/or register chipmay use this retry mechanism to signal and communicate to the hostcomputer.

Hot-Swap and Hot-Plug

In computers used as servers, it is often desired to be able to add orremove memory while the computer is still operating. Such is the case ifthe computer is being used to run an application, such as a web server,that must be continuously operational. The ability to add or removememory in this fashion is called memory hot-plug or hot-swap. Computersthat provide the ability to hot-plug or hot-swap memory use veryexpensive and complicated memory controllers and ancillary hardware,such as latches, programmable control circuits, microcontrollers, aswell as additional components such as latches, indicators, switches, andrelays.

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful hot-swap and hot plug capabilities may beimplemented.

For example, using intelligent buffer and/or register chips on a memorymodule, it is possible to incorporate some or all of the controlcircuits that enable memory hot-swap in these chips.

In conventional memory systems, hot-swap is possible by addingadditional memory modules. Using modules with intelligent buffer and/orintelligent register chips, hot-swap may be achieved by adding DRAM tothe memory module directly without the use of expensive chips andcircuits on the motherboard. In the embodiment shown in FIG. 8A, it ispossible to implement hot-swap by adding further DRAMs to the memorystack. In another implementation as shown in FIG. 8B, hot-swap can beimplemented by providing sockets on the memory module that can acceptDRAM chips or stacks of DRAM chips (with or without intelligent bufferchips). In still another implementation as shown in FIG. 8C, hot-swapcan be implemented by providing a socket on the memory module that canaccept another memory module, thus allowing the memory module to beexpanded in a hot-swap manner.

Redundant Paths

In computers that are used as servers, it is essential that allcomponents have high reliability. Increased reliability may be achievedby a number of methods. One method to increase reliability is to useredundancy. If a failure occurs, a redundant component, path or functioncan take the place of a failure.

In a memory module that includes intelligent register and/or intelligentbuffer chips, extensive datapath redundancy capabilities may beimplemented.

For example, intelligent register and/or intelligent buffer chips cancontain multiple paths that act as redundant paths in the face offailure. An intelligent buffer or register chip can perform a logicalfunction that improves some metric of performance or implements some RASfeature on a memory module, for example. Examples of such features wouldinclude the Intelligent Scrubbing or Autonomous Refresh features,described elsewhere in this specification. If the logic on theintelligent register and/or intelligent buffer chips that implementsthese features should fail, an alternative or bypass path may beswitched in that replaces the failed logic.

Autonomous Refresh

Most computers use DRAM as the memory technology in their memory system.The memory cells used in DRAM are volatile. A volatile memory cell willlose the data that it stores unless it is periodically refreshed. Thisperiodic refresh is typically performed through the command of anexternal memory controller. If the computer fails in such a way that thememory controller cannot or does not institute refresh commands, thendata will be lost.

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful autonomous refresh capabilities may beimplemented.

For example, the intelligent buffer chip attached to a stack of DRAMchips can detect that a required refresh operation has not beenperformed within a certain time due to the failure of the memorycontroller or for other reasons. The time intervals in which refreshshould be performed are known and specific to each type of DRAM. In thisevent, the intelligent buffer chip can take over the refresh function.The memory module is thus capable of performing autonomous refresh.

Intelligent Scrubbing

In computers used as servers, the memory controller may have the abilityto scrub the memory system to improve reliability. Such a memorycontroller includes a scrub engine that performs reads, traversingacross the memory system deliberately seeking out errors. This processis called “patrol scrubbing” or just “scrubbing.” In the case of asingle-bit correctable error, this scrub engine detects, logs, andcorrects the data. For any uncorrectable errors detected, the scrubengine logs the failure, and the computer may take further actions. Bothtypes of errors are reported using mechanisms that are underconfiguration control. The scrub engine can also perform writes known as“demand scrub” writes or “demand scrubbing” when correctable errors arefound during normal operation. Enabling demand scrubbing allows thememory controller to write back the corrected data after a memory read,if a correctable memory error is detected. Otherwise, if a subsequentread to the same memory location were performed without demandscrubbing, the memory controller would continue to detect the samecorrectable error. Depending on how the computer tracks errors in thememory system, this might result in the computer believing that thememory module is failing or has failed. For transient errors, demandscrubbing will thus prevent any subsequent correctable errors after thefirst error. Demand scrubbing provides protection against and permitsdetection of the deterioration of memory errors from correctable touncorrectable.

In a memory module that includes intelligent register and/or intelligentbuffer chips, more powerful and more intelligent scrubbing capabilitiesmay be implemented.

For example, an intelligent register chip or intelligent buffer chip mayperform patrol scrubbing and demand scrubbing autonomously without thehelp, support or direction of an external memory controller. Thefunctions that control scrubbing may be integrated into intelligentregister and/or buffer chips on the memory module. The computer cancontrol and configure such autonomous scrubbing operations on a memorymodule either through inline or out-of-band communications that aredescribed elsewhere in this specification.

Parity Protected Paths

In computers used as servers, it is often required to increase thereliability of the memory system by providing data protection throughoutthe memory system. Typically, data protection is provided by addingredundant information to a data word in a number of ways. As previouslydescribed herein, in one well-known method, called parity protection, asimple code is created by adding one or more extra bits, known as paritybits, to the data word. This simple parity code is capable of detectinga single bit error. In another well-known method, called ECC protection,a more complex code is created by adding ECC bits to the data word. ECCprotection is typically capable of detecting and correcting single-biterrors and detecting, but not correcting, double-bit errors.

These protection schemes may be applied to computation data. Computationdata is data that is being written to and read from the memory system.The protection schemes may also be applied to the control information,memory addresses for example, that are used to control the behavior ofthe memory system.

In some computers, parity or ECC protection is used for computationdata. In some computers, parity protection is also used to protectcontrol information as it flows between the memory controller and thememory module. The parity protection on the control information onlyextends as far as the bus between the memory controller and the memorymodule, however, as current register and buffer chips are notintelligent enough to extend the protection any further.

In a memory module that includes intelligent register and/or intelligentbuffer chips, advanced parity protection coverage may be implemented.

For example, as shown in FIG. 9A, a memory module that includesintelligent buffer and/or register chips, the control paths (those pathsthat involve control information, such as memory address, clocks andcontrol signals and so forth) may be protected using additional paritysignals to ECC protect any group of control path signals in part or inits entirety. Address parity signals 915 computed from the signals ofthe address bus 916, for example, may be carried all the way through thecombination of any intelligent register 902 and/or intelligent bufferchips 907A-907D, including any logic functions or manipulations that areapplied to the address or other control information.

Although the intelligent buffer chips 907A-907D are shown in FIG. 9A asconnected directly to the intelligent register chip 902 and to buffersignals from the intelligent register chip, the same or otherintelligent buffer chips may also be connected to buffer the datasignals. The data signals may or may not be buffered by the intelligentregister chip.

ECC Signaling

The vast majority of computers currently use an electrical bus tocommunicate with their memory system. This bus typically uses one of avery few standard protocols. For example, currently computers use eitherDouble-Data Rate (DDR) or Double-Date Rate 2 (DDR2) protocols tocommunicate between the computer's memory controller and the DRAM on thememory modules that comprise the computer's memory system. Common memorybus protocols, such as DDR, have limited signaling capabilities. Themain purpose of these protocols is to communicate or transfer databetween computer and the memory system. The protocols are not designedto provide and are not capable of providing a path for otherinformation, such as information on different types of errors that mayoccur in the memory module, to flow between memory system and thecomputer.

It is common in computers used as servers to provide a memory controllerthat is capable of detecting and correcting certain types of errors. Themost common type of detection and correction uses a well-known type ofError Correcting Code (ECC). The most common type of ECC allows a singlebit error to be detected and corrected and a double-bit error to bedetected, but not corrected. Again, the ECC adds a certain number ofextra bits, the ECC bits, to a data word when it is written to thememory system. By examining these extra bits when the data word is read,the memory controller can determine if an error has occurred.

In a memory module that includes intelligent register and/or intelligentbuffer chips, a flexible error signaling capability may be implemented.

For example, as shown in FIG. 9, if an error occurs in the memorymodule, an intelligent register and/or buffer chip may deliberatelycreate an ECC error on the data parity signals 917 in order to signalthis event to the computer. This deliberate ECC error may be created byusing a known fixed, hard-wired or stored bad data word plus ECC bits,or a bad data word plus ECC bits can be constructed by the intelligentregister and/or buffer chip. Carrying this concept to a memory subsystemthat includes one or more intelligent register chips and or one or moreintelligent buffer chips, the parity signals 909, 911, and 913 are shownimplemented for signals 908, 910, and 912. Such parity signals can beimplemented optionally for all or some, or none of the signals of amemory module.

This signaling scheme using deliberate ECC errors can be used for otherpurposes. It is very often required to have the ability to request apause in a bus protocol scheme. The DDR and other common memory busprotocols used today do not contain such a desirable mechanism. If theintelligent buffer chips and/or register chips wish to instruct thememory controller to wait or pause, then an ECC error can bedeliberately generated. This will cause the computer to pause and thentypically retry the failing read. If the memory module is then able toproceed, the retried read can be allowed to proceed normally and thecomputer will then, in turn, resume normal operation.

Sideband and Inline Signaling

Also, as shown in FIG. 9, a memory module that includes intelligentbuffer and/or register chips, may communicate with an optional SerialPresence Detect (SPD) 920. The SPD may be in communication with the hostthrough the SPD interface 922 and may be connected to any combination ofany intelligent register 902 and/or any intelligent buffer chips907A-907D. The aforementioned combination implements one or more datasources that can program and/or read the SPD in addition to the host.Such connectivity with the SPD provides the mechanism to performcommunication between the host and memory module in order to transferinformation about memory module errors (to improve Reliability andServiceability features, for example). Another use of the SPD is toprogram the intelligent features of the buffer and/or register chips,such as latency, timing or other emulation features. One advantage ofusing the SPD as an intermediary to perform communication betweenintelligent buffer and/or register chips with the host is that astandard mechanism already exists to use the SPD and host to exchangeinformation about standard memory module timing parameters.

The SPD is a small, typically 256-byte, 8-pin EEPROM chip mounted on amemory module. The SPD typically contains information on the speed,size, addressing mode and various timing parameters of the memory moduleand its component DRAMs. The SPD information is used by the computer'smemory controller to access the memory module.

The SPD is divided into locked and unlocked areas. The memory controller(or other chips connected to the SPD) can write SPD data only onunlocked (write-enabled) DIMM EEPROMs. The SPD can be locked viasoftware (using a BIOS write protect) or using hardware writeprotection. The SPD can thus also be used as a form of sidebandsignaling mechanism between the memory module and the memory controller.

In a memory module that includes intelligent register and/or intelligentbuffer chips, extensive sideband as well as in-band or inline signalingcapabilities may be implemented and used for various RAS functions, forexample.

More specifically, the memory controller can write into the unlockedarea of the SPD and the intelligent buffer and/or register chips on thememory module can read this information. It is also possible for theintelligent buffer and/or register chips on the memory module to writeinto the SPD and the memory controller can read this information. In asimilar fashion, the intelligent buffer and/or register chips on thememory module can use the SPD to read and write between themselves. Theinformation may be data on weak or failed memory cells, error, statusinformation, temperature or other information.

An exemplary use of a communication channel (or sideband bus) betweenbuffers or between buffers and register chips is to communicateinformation from one (or more) intelligent register chip(s) to one (ormore) intelligent buffer chip(s).

In exemplary embodiments, control information communicated using thesideband bus 908 between intelligent register 902 and intelligent bufferchip(s) 907A-907D may include information such as the direction of dataflow (to or from the buffer chips), and the configuration of the on-dietermination resistance value (set by a mode register write command). Asshown in the generalized example 900 of FIG. 9B, the data flow directionon the intelligent buffer chip(s) may be set by a “select port N, bytelane Z” command sent by the intelligent register via the sideband bus,where select 950 indicates the direction of data flow (for a read or awrite), N 951 is the Port ID for one of the multiple data portsbelonging to the intelligent buffer chip(s), and Z 952 would be either 0or 1 for a buffer chip with two byte lanes per port. The bit field 953is generalized for illustration only, and any of the fields 950, 951,952 may be used to carry different information, and may be shorter orlonger as required by the characteristics of the data.

The intelligent register chip(s) use(s) the sideband signal to propagatecontrol information to the multiple intelligent buffer chip(s). However,there may be a limited numbers of pins and encodings used to deliver theneeded control information. In this case, the sideband control signalsmay be transmitted by intelligent register(s) to intelligent bufferchip(s) in the form of a fixed-format command packet. Such a commandpacket be may two cycles long, for example. In the first cycle, acommand type 960 may be transmitted. In the second cycle, the value 961associated with the specific command may be transmitted. In oneembodiment, the sideband command types and encodings to direct data flowor to direct Mode Register Write settings to multiple intelligent bufferchip(s) can be defined as follows (as an example, the command encodingfor the command type 960 for presentation on the sideband bus in thefirst cycle is shown in parenthesis):

Null operation, NOP (000)

Read byte-lane 0 (001)

Write byte-lane 0 (010)

Update Mode Register Zero MRO (011)

Write to both byte lanes 0 and 1 (100)

Read byte-lane 1 (101)

Write byte-lane 1 (110)

Update Extended Mode Register One EMR1 (111)

The second cycle contains values associated with the command in thefirst cycle.

There may be many uses for such signaling. Thus, for example, as shownin FIG. 9D if the bi-directional multiplexer/de-multiplexer onintelligent buffer chip(s) is a four-port-to-one-port structure, thePort IDs would range from 0 to 3 to indicate the path of data flow forread operations or write operations. The Port IDs may be encoded asbinary values on the sideband bus as Cmd[1:0] 962 in the second cycle ofthe sideband bus protocol (for read and write commands).

Other uses of these signals may perform additional features. Thus, forexample, a look-aside buffer (or LAB) may used to allow the substitutionof data from known-good memory bits in the buffer chips for data fromknown-bad memory cells in the DRAM. In this case the intelligent bufferchip may have to be informed to substitute data from a LAB. This actionmay be performed using a command and data on the sideband bus asfollows. The highest order bit of the sideband bus Cmd[2] 963 may usedto indicate a LAB. In the case that the sideband bus Cmd[2] may indicatea LAB hit on a read command, Intelligent buffer chip(s) may then takedata from a LAB and drive it back to the memory controller. In the casethat the sideband bus Cmd[2] indicates a LAB hit on a write command,Intelligent buffer chip(s) may take the data from the memory controllerand write it into the LAB. In the case that the sideband bus Cmd[2] doesnot indicate a LAB hit, reads and writes may be performed to DRAMdevices on the indicated Port IDs.

Still another use as depicted in FIG. 9D of the sideband signal may beto transfer Mode Register commands sent by the memory controller to theproper destination, possibly with (programmable) modifications. In theabove example command set, two commands have been set aside to updateMode Registers.

One example of such a register mode command is to propagate an MROcommand, such as burst ordering, to the intelligent buffer chip(s). Forexample, Mode Register MRO bit A[3] 964 sets the Burst Type. In thiscase the intelligent register(s) may use the sideband bus to instructthe intelligent buffer chip(s) to pass the burst type (through thesignal group 906) to the DRAM as specified by the memory controller. Asanother example, Mode Register MRO bit A[2:0] sets the Burst Length 965.In this case, in one configuration of memory module, the intelligentregister(s) may use the sideband bus to instruct the intelligent bufferchip(s) to always write '010 (corresponding to a setting of burst lengthequal to four or BL4) to the DRAM. In another configuration of memorymodule, if the memory controller had asserted '011, then the intelligentregister(s) must emulate the BL8 column access with two BL4 columnaccesses.

In yet another example of this type sideband bus use, the sideband busmay be used to modify (possibly under programmable control) the valuesto be written to Mode Registers. For example, one Extended Mode RegisterEMR1 command controls termination resistor values. This command sets theRtt (termination resistor) values for ODT (on-die termination), and inone embodiment the intelligent register chip(s) may override existingvalues in the A[6] A[2] bits in EMR1 with '00 to disable ODT on the DRAMdevices, and propagate the expected ODT value to the intelligent bufferchip(s) via the sideband bus.

In another example, the sideband signal may be used to modify thebehavior of the intelligent buffer chip(s). For example, the sidebandsignal may be used to reduce the power consumption of the intelligentbuffer chip(s) in certain modes of operation. For example, anotherExtended Mode Register EMR1 command controls the behavior of the DRAMoutput buffers using the Qoff command. In one embodiment, theintelligent register chip(s) may respect the Qoff request meaning theDRAM output buffers should be disabled. The intelligent register chip(s)may then pass through this EMR1 Qoff request to the DRAM devices and mayalso send a sideband bus signal to one or more of the intelligent bufferchip(s) to turn off their output buffers also—in order to enable IDDmeasurement or to reduce power for example. When the Qoff bit it set,the intelligent register chip(s) may also disable all intelligent bufferchip(s) in the system.

Additional uses envisioned for the communication between intelligentregisters and intelligent buffers through side-band or inline signalinginclude:

-   -   a. All conceivable translation and mapping functions performed        on the Data coming into the Intelligent Register 902. A        ‘function’ in this case should go beyond merely repeating input        signals at the outputs.    -   b. All conceivable translation and mapping functions performed        on the Address and Control signals coming into the Intelligent        Register 902. A ‘function’ in this case should go beyond merely        repeating input signals at the outputs.    -   c. Uses of any and every signal originating from the DRAM going        to the Intelligent Register or intelligent buffer.    -   d. Use of any first signal that is the result of the combination        of a second signal and any data stored in non-volatile storage        (e.g. SPD) where such first signal is communicated to one or        more intelligent buffers 907.    -   e. Clock and delay circuits inside the Intelligent Register or        intelligent buffer. For example, one or more intelligent buffers        can be used to de-skew data output from the DRAM.

Still more uses envisioned for the communication between intelligentregisters and intelligent buffers through sideband or inline signalinginclude using the sideband as a time-domain multiplexed address bus.That is, rather than routing multiple physical address busses from theintelligent register to each of the DRAMs (through an intelligentbuffer), a single physical sideband shared between a group ofintelligent buffers can be implemented. Using a multi-cycle command &value technique or other intelligent register to intelligent buffercommunication techniques described elsewhere in this specification, adifferent address can be communicated to each intelligent buffer, andthen temporally aligned by the intelligent buffer such that the dataresulting from (or presented to) the DRAMs is temporally aligned as agroup.

Bypass and Data Recovery

In a computer that contains a memory system, information that iscurrently being used for computation is stored in the memory modulesthat comprise a memory system. If there is a failure anywhere in thecomputer, the data stored in the memory system is at risk to be lost. Inparticular, if there is a failure in the memory controller, theconnections between memory controller and the memory modules, or in anychips that are between the memory controller and the DRAM chips on thememory modules, it may be impossible to retain and retrieve data in thememory system. This mode of failure occurs because there is noredundancy or failover in the datapath between the memory controller andDRAM. A particularly weak point of failure in a typical DIMM lies in theregister and buffer chips that pass information to and from the DRAMchips. For example, in an FB-DIMM, there is an AMB chip. If the AMB chipon an FB-DIMM fails, it is not possible to retrieve data from the DRAMon that FB-DIMM.

In a memory module that includes intelligent register and/or intelligentbuffer chips, more powerful memory buffer bypass and data recoverycapabilities may be implemented.

As an example, in a memory module that uses an intelligent buffer orintelligent register chip, it is possible to provide an alternativememory datapath or read mechanism that will allow the computer torecover data despite a failure. For example, the alternative datapathcan be provided using the SMBus or I2C bus that is typically used toread and write to the SPD on the memory module. In this case the SMBusor I2C bus is also connected to the intelligent buffer and/or registerchips that are connected to the DRAM on the memory module. Such analternative datapath is slower than the normal memory datapath, but ismore robust and provides a mechanism to retrieve data in an emergencyshould a failure occur.

In addition, if the memory module is also capable of autonomous refresh,which is described elsewhere in this specification, the data may stillbe retrieved from a failed or failing memory module or entire memorysystem, even under conditions where the computer has essentially ceasedto function, due to perhaps multiple failures. Provided that power isstill being applied to the memory module (possibly by an emergencysupply in the event of several failures in the computer), the autonomousrefresh will keep the data in each memory module. If the normal memorydatapath has also failed, the alternative memory datapath through theintelligent register and/or buffer chips can still be used to retrievedata. Even if the computer has failed to the extent that the computercannot or is not capable of reading the data, an external device can beconnect to a shared bus such as the SMBus or I2C bus used as thealternative memory datapath.

Control at Sub-DIMM Level

In a memory module that includes intelligent register and/or intelligentbuffer chips, powerful temperature monitoring and control capabilitiesmay be implemented, as described elsewhere in this specification. Inaddition, in a memory module that includes intelligent register and/orintelligent buffer chips, extensive control capabilities, includingthermal and power control at the sub-DIMM level, that improvereliability, for example, may be implemented.

As an example, one particular DRAM on a memory module may be subjectedto increased access relative to all the other DRAM components on thememory module. This increased access may lead to excessive thermaldissipation in the DRAM and require access to be reduced by throttlingperformance. In a memory module that includes intelligent registerand/or intelligent buffer chips, this increased access pattern may bedetected and the throttling performed at a finer level of granularity.Using the intelligent register and/or intelligent buffer chips,throttling at the level of the DIMM, a rank, a stack of DRAMs, or evenan individual DRAM may be performed.

In addition, by using intelligent buffer and/or register chips, thethrottling or thermal control or regulation may be performed. Forexample the intelligent buffer and/or register chips can use the ChipSelect, Clock Enable, or other control signals to regulate and controlthe operation of the DIMM, a rank, a stack of DRAMs, or individual DRAMchips. Self-Test

Memory modules used in a memory system may form the most expensivecomponent of the computer. The largest current size of memory module is4 GB (a GB or gigabyte is 1 billion bytes or 8 billion bits) and such amemory module costs several thousands of dollars. In a computer thatuses several of these memory modules (it is not uncommon to have 64 GBof memory in a computer), the total cost of the memory may far exceedthe cost of the computer.

In memory systems, it is thus exceedingly important to be able tothoroughly test the memory modules and not discard memory modulesbecause of failures that can be circumvented or repaired.

In a memory module that includes intelligent register and/or intelligentbuffer chips, extensive DRAM advanced self-test capabilities may beimplemented.

For example, an intelligent register chip on a memory module may performself-test functions by reading and writing to the DRAM chips on thememory module, either directly or through attached intelligent bufferchips. The self-test functions can include writing and reading fixedpatterns, as is commonly done using an external memory controller. As aresult of the self-test, the intelligent register chip may indicatesuccess or failure using an LED, as described elsewhere in thisspecification. As a result of the self-test, the intelligent register orintelligent buffer chips may store information about the failures. Thisstored information may then be used to re-map or map out the defectivememory cells, as described elsewhere in this specification.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof. For example, persons skilled inthe art will appreciate that the features and functionalities describedherein may be implemented using either an intelligent register chip, anintelligent buffer chip, both intelligent chips, or any combinationthereof. The scope of the present invention is thereof determined by theclaims that follow.

1. (canceled)
 2. A memory module comprising: a plurality of memory chipscomprising a first memory chip and a second memory chip; a plurality ofintelligent buffer chips comprising: a first intelligent buffer chipcoupled to the first memory chip, wherein the first intelligent bufferchip includes a first temperature monitoring device configured toperform temperature measurements of the first memory chip; and adistinct second intelligent buffer chip coupled to the second memorychip, wherein the second intelligent buffer chip includes a secondtemperature monitoring device configured to perform temperaturemeasurements of the second memory chip; an intelligent register chipcoupled to a memory controller; and a sideband bus coupling theintelligent register chip and the plurality of intelligent buffer chips,wherein the intelligent register chip is configured to receive (i) fromthe first intelligent buffer chip via the sideband bus, first datarepresenting the temperature measurements of the first memory chip and(ii) from the second intelligent buffer chip via the sideband bus,second data representing the temperature measurements of the secondmemory chip.
 3. The memory module of claim 2, further comprising: aserial presence detect (SPD) chip coupled to the memory controller andthe intelligent register chip, the SPD chip configured to: receive thefirst data and the second data from the intelligent register chip; storethe first data and the second data; and provide the first data and thesecond data to the memory controller.
 4. The memory module of claim 2,wherein the first intelligent buffer chip and the first memory chip forma first stack on the memory module, and the second intelligent bufferchip and the second memory chip form a second stack on the memorymodule.
 5. The memory module of claim 2, wherein the intelligentregister chip is configured to issue commands via the sideband bus toreduce power consumption of the first intelligent buffer chip.
 6. Thememory module of claim 5, wherein the intelligent register chip isconfigured to turn off an output buffer of the first intelligent bufferchip.
 7. The memory module of claim 2, wherein the first intelligentbuffer chip is configured to control a light-emitting diode to indicatea failure of the memory module.
 8. The memory module of claim 2, whereinthe intelligent register chip is configured to perform a self-test onthe memory module.
 9. A memory module comprising: a plurality of memorychips comprising a first memory chip and a second memory chip; aplurality of intelligent buffer chips comprising: a first intelligentbuffer chip coupled to the first memory chip, wherein the firstintelligent buffer chip includes a first temperature monitoring deviceconfigured to perform temperature measurements of the first memory chip;and a distinct second intelligent buffer chip coupled to the secondmemory chip, wherein the second intelligent buffer chip includes asecond temperature monitoring device configured to perform temperaturemeasurements of the second memory chip; an intelligent register chipcoupled to a memory controller; a sideband bus coupling the intelligentregister chip and the plurality of intelligent buffer chips; a firstdata bus coupling the intelligent register chip and the first memorychip; and a second data bus coupling the intelligent register chip andthe second memory chip, wherein the intelligent register chip isconfigured to receive (i) from the first intelligent buffer chip via thesideband bus, first data representing the temperature measurements ofthe first memory chip and (ii) from the second intelligent buffer chipvia the sideband bus, second data representing the temperaturemeasurements of the second memory chip.
 10. The memory module of claim9, further comprising: a serial presence detect (SPD) chip coupled tothe memory controller and the intelligent register chip, the SPD chipconfigured to: receive the first data and the second data from theintelligent register chip; store the first data and the second data; andprovide the first data and the second data to the memory controller. 11.The memory module of claim 9, wherein the first intelligent buffer chipand the first memory chip form a first stack on the memory module, andthe second intelligent buffer chip and the second memory chip form asecond stack on the memory module.
 12. The memory module of claim 9,wherein the intelligent register chip is configured to issue commandsvia the sideband bus to reduce power consumption of the firstintelligent buffer chip.
 13. The memory module of claim 12, wherein theintelligent register chip is configured to turn off an output buffer ofthe first intelligent buffer chip.
 14. The memory module of claim 9,wherein the first intelligent buffer chip is configured to control alight-emitting diode to indicate a failure of the memory module.
 15. Thememory module of claim 9, wherein the intelligent register chip isconfigured to perform a self-test on the memory module.
 16. A memorymodule comprising: a plurality of memory chips comprising a first memorychip and a second memory chip; a plurality of intelligent buffer chipscomprising: a first intelligent buffer chip coupled to the first memorychip, wherein the first intelligent buffer chip includes a firsttemperature monitoring device configured to perform temperaturemeasurements of the first memory chip; and a distinct second intelligentbuffer chip coupled to the second memory chip, wherein the secondintelligent buffer chip includes a second temperature monitoring deviceconfigured to perform temperature measurements of the second memorychip; an intelligent register chip coupled to a memory controller; asideband bus coupling the intelligent register chip and the plurality ofintelligent buffer chips; a first data bus coupling the memorycontroller and the first memory chip; and a second data bus coupling thememory controller and the second memory chip, wherein the intelligentregister chip is configured to receive (i) from the first intelligentbuffer chip via the sideband bus, first data representing thetemperature measurements of the first memory chip and (ii) from thesecond intelligent buffer chip via the sideband bus, second datarepresenting the temperature measurements of the second memory chip. 17.The memory module of claim 16, further comprising: a serial presencedetect (SPD) chip coupled to the memory controller and the intelligentregister chip, the SPD chip configured to: receive the first data andthe second data from the intelligent register chip; store the first dataand the second data; and provide the first data and the second data tothe memory controller.
 18. The memory module of claim 16, wherein thefirst intelligent buffer chip and the first memory chip form a firststack on the memory module, and the second intelligent buffer chip andthe second memory chip form a second stack on the memory module.
 19. Thememory module of claim 16, wherein the intelligent register chip isconfigured to issue commands via the sideband bus to reduce powerconsumption of the first intelligent buffer chip.
 20. The memory moduleof claim 19, wherein the intelligent register chip is configured to turnoff an output buffer of the first intelligent buffer chip.
 21. Thememory module of claim 16, wherein the first intelligent buffer chip isconfigured to control a light-emitting diode to indicate a failure ofthe memory module.